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USE OF NORMAL MESHES IN THREE-DIMENSIONAL IMAGING 



Background 



Three-dimensional imaging often requires three scalar 
functions such as x, y, and z coordinates. These coordinates 
define parameters of the surface so that the surface can be 
visualized as a three dimensional image. 



The present application teaches a new kind of way of 
describing a three dimensional surface. The description is 
called a "normal mesh". The mesh has information which 
defines information relative to a special tangent plane. 

In one embodiment, the normal mesh is defined as a normal 
offset from a coarser version. The mesh can be stored with a 
single float per vertex, thus reducing the amount of 
information which needs to be stored. 



Summary 
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Brief Description of the Drawings 
These and other aspects will now be described in 
detail with reference to the accompanying drawings, in which: 
5 Figure 1 shows how a smooth surface of three dimensions 

can be described in terms of single variable scalars; 
Figure 2 shows a polyline; 

Figure 3 shows construction of a normal polyline; 

Figure 4A shows a flowchart of forming a polyline; 
IP 10 Figure 4B shows a flowchart of overall operation of 

ri' compressing the surface; 

1% Figures 5A-5F show the various stages of compressing a 

w 

? sample surface, here a molecule; 

o 

'""i Figure 6 shows a based domain vertext repositioning; 

15 Figure 7 shows a piercing operation; 

Figure 8 shows a face splitting operation to obtain 
additional surface detail; and 

Figure 9 shows a result of applying a naive piercing 
procedure . 

2 0 Detailed Description 

Figure 1 shows how a smooth surface 100 can be locally 
described by single variable scalar height functions, hi, h 2 , 
h 3 , h 4 over a tangent plane 110. When considered this way, the 
three dimensional information for the smooth surface 100 is 
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contained only in this single dimension h: the height over 
the tangent plane. In practice, this approximation only works 
inf initesimally . However, it may provide interesting 
information . 

Surfaces are often approximated using a triangle mesh. 
However, this description may lose structural assumption that 
are inherent in the actual surface. For example, some of the 
smoothness assumption that one can make in an actual surface 
may be lost in the triangle mesh. Hence, the triangle mesh 
has inherent redundancy. 

For a given smooth shape, different parameterizations may 
still keep the geometry the same. In defining a mesh, the 
present application notices that infinitesimal tangential 
motion of a vertex does not change the geometry. However, 
moving in the normal direction does change the geometry. 

The normal meshes which are described herein require only 
a single scalar value per vertex. This is may be done using a 
multiresolution and local frame. A hierarchical 
representation provides that all detail coefficients expressed 
in these frames are scalar. The parameter may be a normal 
component, for example. In the context of compression, for 
example, this allows parameter information to be predicted and 
confines residual error to the normal direction. 
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A curve in a plane can be defined by a pair of parametric 
functions . 

S(t) = (x(t),y(t)) with t g [0,1] . In the present embodiment, 
polylines may be used to approximate curves. 

Let £(p,p') be the linear segment between the points p and 
p\ A polyline multiresolution approximation is made by 
sampling the curve at points Sj, k where Sj /k = Sj+i,2k and 
defining the jth level approximation as 



To move from Lj to Lj+i, the points Sj+i, 2 k+i are inserted (Figure 
2, left). Clearly this requires two scalars: the two 
coordinates of Sj+i,2k+i- Alternatively one could compute the 
difference Sj+i,2k+i ~ m between the new point and some predicted 
point m, for example, the midpoint of the neighboring points 
s j/k and Sj,k + i. This detail has a tangential component m- b and 
a normal component b-Sj+i,2k+i • The normal component represents 
the geometric information while the tangential component the 
parameter information. 

Figure' 2 shows removing one point + 1,2* + 1>) in a polyline 
multiresolution and recording the difference with the midpoint 
m. On the left a general polyline where the detail has both a 
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normal and a tangential component. On the right is a normal 
polyline where the detail is purely normal. 

Polylines can hence be described with one scalar per 
point if the parameter information is always zero, i.e., b = 
m, in Figure 2B. If the triangle Sj /k , Sj+i,2k+if s j,k+i is 
Isosceles, there is no parameter information. 

Hence a polyline is "normal" if a multiresolution 
structure exists where every removed point forms an Isosceles 
triangle with its neighbors. Then there is zero parameter 
information and the polyline can be represented with one 
scalar per point, namely the normal component of the 
associated detail . 

Figure 3 shows construction of a normal polyline. We 
start with the coarsest level and each time check where the 
normal to the midpoint crosses the curve. For simplicity only 
the indices of the Sj, k points are shown and only certain 
segments are subdivided. The polyline ( 0 , 0 ) - ( 2 , 1 ) - ( 3 , 3 ) - ( 1 , 1 ) - 
(0,1) is determined by its endpoints and three scalars, the 
heights of the Isosceles triangles. 

For a general polyline, the removed triangles are hardly 
ever exactly Isosceles, and the polyline hence not normal. A 
normal polyline approximation for any continuous curve using 
the following techniques. The easiest is to start building 
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Isosceles triangles from the coarsest level. The operation 
starts with the first base 1 (s 0 ,o/ so, i) , see Figure 3. Next, 
its midpoint is taken. A determination is made of where the 
normal direction crosses the curve. Because the curve is 
continuous, there has to be at least one such point. If there 
are multiple points, any one point can be selected. This point 
can be labeled as si,i. The first triangle is defined using 
this point. Now this process is repeated. Each time Sj + i f2 jc+i is 
found where the normal to the midpoint of Sj, k and Sj /k+ i crosses 
the curve. Thus, any continuous curve can be approximated 
arbitrarily closely with a normal polyline. The result is a 
series of polylines Lj, all of which are normal with respect 
to midpoint prediction. Effectively each level is 
parameterized with respect to the one coarser level. Because 
the polylines are normal, only a single scalar value, the 
normal component, needs to be recorded for each point. These 
polylines may have no parameter information. 

One can also consider normal polylines with respect to 
other predictors. A base point and normal estimate can be 
produced using the well known 4 point rule. Any predictor 
which only depends on the coarser level is allowed. Irregular 
schemes described in Daubechies, I., Guskov, I., and Sweldens, 
W. Regularity of Irregular Subdivision. Constr. Approx. 15 
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(1999), 381-426. can also be used. Levels may be built by 
downsampling every other point, or using any other ordering. 

Describing this in terms of further generality, a 
polyline is normal if a removal order of the points exist such 
that each removed point lies in the normal direction from a 
base point f where the normal direction and base point only 
depend on the remaining points. 

Hence a normal polyline may be completely determined by a 
single scalar component per vertex. 

Normal polylines are closely related to certain well 
known fractal curves such as the Koch Snowflake. The normal 
coefficients can be thought of as a piecewise linear wavelet 
transform of the original curve. Because the tangential 
components are always zero, there may be half as many wavelet 
coefficients as the original scalar coefficients. The 
wavelets have their usual decorrelation properties. 

A triangle mesh M is a pair (P,K) , where P is a set of N 

point positions P = {/> = 0Y,,F„Z,) e R 3 \ mid 1 < i < N) , and K is an 

abstract simplicial complex which contains all the 

topological, i.e., adjacency information. The complex K is a 

set of subsets of {1,...,N}. These subsets come in three types: 

vertices {i}, edges {i,j}, and faces {i,j,k}. Two vertices i 
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and j are neighbors if {i,j}eE. The 1-ring neighbors of a 
vertex I form a set V(i) = {j/mid{i,j}e E} . 

Definition of normal triangle meshes may be inspired by 
the curve case. Consider a hierarchy of triangle meshes Mj 
built using mesh simplification with vertex removals. These 
meshes are nested in the sense that Pj c P J+l . Take a removed 

vertex P i sPj +l \P J . For the mesh to be normal we need to be 

able to find a base point b and normal direction N that only 

depend on P., so that P.~ b lies in the direction N. This leads 

to the definition that a mesh M is normal in case a sequence 
of vertex removals exists so that each removed vertex lies on 
a line defined by a base point and normal direction which only 
depends on the remaining vertices. 

Thus a normal mesh can be described by a small base 
domain and one scalar coefficient per vertex. 

A mesh in general is not normal, just as a curve is in 
general not normal. The present application therefore uses a 
special kind of mesh, called a semi-regular mesh. The semi- 
regular mesh has a connectivity which is formed by successive 
quadrasection of coarse base domain faces. The operation is 
shown in Figure 4 at 440, the operation begins with a coarsest 
level or base domain. If there are no new vertices, the 
operation is complete at 410. For each new vertex determined 
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at 405, a new base point is computed and a normal direction 
are found at 415. A- determination is made of where the line 
defined by the base point and normal intersects the surface 
420. 425 determines how many intersection points exist. If 
only one point exists, it is accepted at 430. In the surface 
situation, there might be no intersection point or many 
intersection points, not all of which are correct. 

If there are no intersection points, control passes to 
the left. A fully normal mesh could not be built from this 
base domain. Therefore, the definition of a normal mesh is 
rearranged to allow a small number of cases where the new 
points do not lie in the normal direction. The technique 
needs to find a suitable non-normal direction in order to 
proceed. 

A smooth parameterization helps define the surface. 
Several parameterization techniques have been proposed 
including MAPS, patch wise relaxation, and specific smoothness 
function, see, Dyn, N., Levin, D., and Gregory, J. A. A 
Butterfly Subdivision Scheme for Surface Interpolation with 
Tension Control. ACM Transactions on Graphics 9, 2 (1990), 
160-169. Eck, M . , DeRose, T., Duchamp, T., Hoppe, H., 
Lounsbery, M., and Stuetzle, W. Multiresolution Analysis of 
Arbitrary Meshes. Proceedings of SIGGRAPH 95 (1995), 173-182. 
; Lee, A. W. F . , Dobkin, D., Sweldens, W., and Schroder, P. 
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Multiresolution Mesh Morphing. Proceedings of SIGGRAPH 99 
(1999), 343-350; Levoy, M. The Digital Michelangelo Project. 
In Proceedings of the 2nd International Conference on 3D 
Digital Imaging and Modeling, October 1999. 

Consider a region R of the mesh homeomorphic to a disc 
that is to be parameterized onto a convex planar region B, 
i.e., find a bijective map u:R^>B. The map u is fixed by a 
boundary condition dR-^dB and minimizes a certain energy 
functional. Several functionals can be used leading to, e.g., 
conformal or harmonic mappings. The disclosed system takes an 
approach based on the parameterization scheme introduced by 
Floater. In short, the function u needs to satisfy the 
following equation in the interior: 



u{Pi) = 



k e V(i) 



a ik u(Pk\ 



(1) 



where V{i) is the 1-ring neighborhood of the vertex i and the 
weights a ik come from the Floater parameterization scheme 

introduced by Floater. The Floater weights is that they are 
always positive, which, combined with the convexity of the 
parametric region, guarantees that no triangle flipping can 
occur within the parametric domain. This is not true in 



10 



f 



Attorn^Mocket No. 06618/580001 



general for harmonic maps which can have negative weights. 
The iterative biconjugate gradient method is used to obtain 
the solution to the system. 

The overall image formation is shown in the flowchart of 
5 Figure 4B. Figure 5 shows a highly detailed and curved model 
of a molecule and these steps. 

1. Mesh simplification: At 4 50, the Garland- 

Heckbert simplification, based on half-edge collapses, is used 
to create a mesh hierarchy (Pj,Kj). We use the coarsest level 
10 {Po f Ko) as an initial guess for our base domain (Qo,K 0 ). The 
first image, shown in Figure 5A, shows an image of the base 
domain for the molecule. Note that this is relatively coarse. 

2. Building an initial net of curves: At 4 60, an initial 
set of curves is defined, to connect the vertices of the base 

15 domain with a net of non intersecting curves on the different 
levels of the mesh simplification hierarchy. This can be done 
using the MAPS parameterization. MAPS uses polar maps to build 
a bijection between a 1-ring and its retriangulation after the 
center vertex is removed. The concatenation of these maps is a 

20 bijective mapping between different levels (Pj,Kj) in the 

hierarchy. The desired curves include the image of the base 
domain edges under this mapping. Because of the bijection, no 
intersection can occur. Note that the curves start and finish 
at a vertex of the base domain. They need not follow the 
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edges of the finer triangulation, i.e., they can cut across 
triangles. These curves define a network of triangular shaped 
patches corresponding to the base domain triangles. Later 
these curves will be adjusted on some intermediate level. 
Again MAPS may be used to propagate these changes to other 
levels. Figure 5B shows these curves for some intermediate 
level of the hierarchy. 

3. Fixing the global vertices: A normal mesh is almost 
completely determined by the base domain. Selection of the 
base domain vertices Qo may reduce the number of non-normal 
vertices to a minimum. The coarsest level of the mesh 
simplification P 0 is only a first guess. 

At 460, the global vertices qi are respositioned with {i} 
g Kb. Constraint is imposed that the qi needs to coincide 
with some vertex p k of the original mesh, but not necessarily 

Pi- 

The repositioning is typically done on some intermediate 
level j . Take a base domain vertex qi shown on the left in 
Figure 6. We build a parameterization from the patches 
incident to vertex qi to a disk in the plane 610, see 
Figure 6. Boundary conditions are assigned using arc length 
parameterization. Parameter coordinates are iteratively 
computed for each level j vertex inside the shaded region. 
The point qi may be replaced with any level point from Pj in 
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the shaded region. The new qi' may be the point of Pj that in 
the parameter domain is closest to the center of the disk. ' 

Once a new position qi' is chosen, the curves can be 
redrawn by taking the inverse mapping of straight lines from 
the new point in the parameter plane. This procedure may be 
iterated. It may alternatively suffice to cycle once through 
all base domain vertices. 

User controlled repositioning may allow the user to 
replace the center vertex with any Pj point in the shaded 
region. Parameterization may be used to recompute the curves 
from that point. 

Figure 5C shows the repositioned vertices. Notice how 
some of them, like the rightmost ones have moved considerably. 

Figure 6 shows base domain vertex repositioning with the 
left showing original patches around qi, middle: parameter 
domain, right: repositioned qi and new patch boundaries. This 
is replaced with the vertex whose parameter coordinate are the 
closest to the center. The inverse mapping (right) is used to 
find the new position qi' and the new curves. 

4. Fixing the global edges: The image of the global 
edges on the finest level will later be the patch boundaries 
of the normal mesh. For this reason, the smoothness of the 
associated curves be improved at the finest level. 4 65 
defines fixing global edges using a procedure similar to Eck, 
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M. , DeRose, T., Duchamp, T., Hoppe, H., Lounsbery, M . , and 
Stuetzle, W. Multiresolution Analysis of Arbitrary Meshes. 
Proceedings of SIGGRAPH 95 (1995), 173-182. For each base 
domain edge {i,k} region formed on the finest level mesh by 
its two incident patches. Let 1 and m be the opposing global 
vertices. A scalar parameter function p within the diamond- 
shaped region of the surface is compiled. The boundary 
condition is set as p (qj = p (q k ) = 0, p (qi) = 1, p (qj = -1, 
with linear variation along the edges. The parameterization 
is compiled and its zero level set is the new curve. One 
could iterate this procedure until convergence but in practice 
one cycle may suffice. The curves of Figure 5D represent the 
result of the curve smoothing on the finest level. 

5. Initial parameterization: Once the global vertices 
and edges are fixed the interior may be filled at 470. This 
is done by computing parameterization of each patch to a 
triangle while keeping the boundary fixed. The parameter 
coordinates from the last stage can serve as a good initial 
guess a smooth global parameterization is shown in the bottom 
left of Figure 5E. Each triangle is given a triangular 
checkerboard texture to illustrate the parameterization. 

Figure 7 shows Upper left: piercing, the Butterfly point 
is s, the surface is pierded at the point q, the 
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parametrically suggested point v lies on the curve separating 
two regions of the mesh. Right: parameter domain, the pierced 
point falls inside the aperture and gets accepted- Lower left: 
the parameterization is adjusted to let the curve pass through 

q- 

6. Piercing: Piercing, at 475, piercing is used to start 
building the actual normal mesh. Figure 7 shows the canonical 
step for a new vertex of the semi-regular mesh to find its 
position on the original mesh. In quadrisection, every edge 
of level j generates a new vertex on level j+1. First, compute 
a base point is computed using interpolating Butterfly 
subdivision as well as an approximation of the normal. This 
defines a straight line. This line may have multiple or no 
intersection points with the original surface. The new vertex 
q may lie halfway along the edge {a, c} with incident 
triangles {a, c, b} and {c, a, d}, see Figure 7. Let the two 
incident patches form the region R. 

Build the straight line L defined by the base point s 
predicted by the Butterfly subdivision rule and the direction 
of the normal computed from the coarser level points. 'All the 
intersection points of L are found with the region R by 
checking all triangles inside. 

If there is no intersection the point v that lies midway 
between the points a and c in the parameter domain is taken: 
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u(v) = (u(a)+u(c)) / 2. This is the same point a standard 
parameterization based remesher would use. 

In the case when there exist several intersections of the 
mesh region R with the piercing line L we choose the 
intersection point that is closest to the point u(v) in the 
parameter domain. Let us denote by u(q) the parametric 
coordinates of that piercing point. We accept this point as a 
valid point of the semi-regular mesh if 

||u(q)-u(v) || <k || u(a)-u(v) || , 

where k is an "aperture" parameter that specifies how 
much the parameter value of a pierced point is allowed to 
deviate from the center of the diamond. Otherwise, the 
piercing point is rejected and the mesh takes the point with 
the parameter value u(v). 

7. Adjusting the parameterization: Once there is a new 
piercing point, we need to adjust the parameterization to 
reflect this at 480. Essentially, the adjusted 
parameterization u should be such that the piercing point has 
the parameters u(v) = :u(q). When imposing such an isolated 
point constraint on the parameterization, there is no 
mathematical guarantee against flipping. Hence a new 
piecewise linear curve through u (q) in the parameter domain is 
prepared. This gives a new curve on the surface which passes 
through q, see Figure 7. The parameterization for each of the 
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patches onto a triangle is separately computed. A piecewise 
linear boundary condition, with the half point at q on the 
common edge, is produced. 

When all the new midpoints for the edges of a face of 
level j are computed, the faces of level j+1 are found. This 
is done by drawing three new curves inside the corresponding 
region of the original mesh, as shown in Figure 8. Before that 
operation happens we need to ensure that a valid 
parameterization is available within the patch. The patch is 
parameterized onto a triangle with three piecewise linear 
boundary conditions each time putting the new points at the 
midpoint. Then the new points are connected in the parameter 
domain which allows us to draw new finer level curves on the 
original mesh. This produces a metamesh similar to ** [ 14 ] 
which replicates the structure of the semi-regular hierarchy 
on the surface of the original. The construction of the semi- 
regular mesh can be done adaptively with the error driven 
procedure from MAPS [15] . An example of parameterization 
adjustment after two levels of adaptive subdivision is shown 
Figure 5F. 

As the parametrization regions become smaller, the 
starting guesses are better and the solver becomes faster. 
Lazy parameter computation may be used, and the relaxation is 



Attorn^Mbcket No. 06618/580001 

run just before we actually need to use parameters for either 
a point location or a surface curve drawing procedure. 
Figure 8 shows a Face split: Quadrisection in the parameter 
plane (left) leads to three new curves within the triangular 
surface region ( right ). The aperture parameter k of the piercing 
procedure provides control over how much of the original 
parameterization is preserved in the final mesh. At k = 0 a 
mesh can be built based entirely based on the original global 
parameterization- At k = 1 a purely normal mesh can be made 
which is independent of the parameterization. The best results 
may be achieved when the aperture was set low (0.2) at the 
coarsest levels, and then increased to (0.6) on finer levels. 
On the very fine levels of the hierarchy, where the geometry 
of the semi-regular meshes closely follows the original 
geometry, a naive piercing procedure without parameter 
adjustment. Figure 9 illustrates such a Naive piercing 
procedure. Clearly, several regions have flipped triangles and 
are self-intersecting. 

Figure 9 shows 4 levels of naive piercing for the torus 
starting from a 102 vertex base mesh. Clearly, there are 
several regions with flipped and self -intersecting triangles. 
The error is about 20 times larger than the true normal mesh. 
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Normal meshes have numerous applications. The following 
are examples. 

Compression Usually a wavelet transform of a standard 
mesh has three components which need to be quantized and 
5 encoded. Information theory tells us that the more non uniform 
the distribution of the coefficients the lower the first order 
entropy. Having 2/3 of the coefficients exactly zero will 
further reduce the bit budget. From an implementation 
viewpoint, the normal mesh coefficients may be connected to 

10 the best known scalar wavelet image compression code. 

Filtering It has been shown that applications such as 
smoothing, enhancement, and denoising can simply be effected 
through a suitable scaling of wavelet coefficients. In a 
normal mesh any such algorithm will run three times as fast. 

15 Also large scaling coefficients in a standard mesh will 
introduce large tangential components leading to flipped 
triangles. 'In a normal mesh this is much less likely to 
happen . 



20 inside patches, across global edges, and around global 

vertices even when the base domain is exceedingly coarse, 

cf. the skull model. The implied parameterizations are highly 

suitable for all types of mapping applications. 



Texturing 



Normal semi-regular meshes are very smooth 
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Rendering Normal maps are a very powerful tool for 
decoration and enhancement of otherwise smooth geometry. In 
particular in the context of bandwidth bottlenecks it is 
attractive to be able to download a normal map into hardware 
and only send smooth coefficient updates for the underlying 
geometry. The normal mesh transform effectively solves the 
associated inverse problem: construct a normal map for a given 
geometry . 

Although only a few embodiments have been disclosed in detail 
above, other modifications are possible. 



20 



